Training and Evaluation of a Spoken Language Understanding System

Introduct ion This paper describes our results on a spoken language application for finding directions. The spoken language system consists of the MIT SUMMIT speech recognition system ([20]) loosely coupled to the UNISYS PUNDIT language understanding system ([9]) with SUMMIT providing the top N candidates (based on acoustic score) to the P U N D I T system. The direction finding capability is provided by an expert system which is also part of the MIT VOYAGER system [18]). 1 One major goal in this research has been to understand issues of training vs. coverage in porting a language understanding system to a new domain. Specifically, we wished to determine how much data it takes to train a spoken language system to a given level of performance for a new domain. We can use the answer to this question in the process of designing data collection tasks to decide how much data to collect. We address a related question, that is, how to quantify the growth of a system as a function of training, in [12]. To explore the relationship of training to coverage, we have developed a methodology to measure coverage of unseen material as a function of training material. Using successive batches of new material, we assessed coverage on a batch of unseen material, then trained on this material until we reached a certain level of coverage, then repeated the experiment on a new batch of material. The system coverage seemed to level off at about 70% coverage of unseen data after 1000 sentences of training data. A second goal was to develop a methodology for automatically tuning a broad-coverage grammar to a new application domain. This approach avoids repeating domain independent grammar development work over again for each new domain. To do this we developed a method for deriving a minimal grammar and a minimal